Interval flow-based inband telemetry

ABSTRACT

This disclosure presents several embodiments of a device configured to perform inband telemetry tracking. The device is configured to store a flow table that tracks metrics of a detected packet flow that passes through the device. More specifically, the flow table tracks several telemetry metrics, each telemetry metric being specific to a tracking time period, and tracking statics based only on packets of the packet flow that were received during that specific time period. At an end of an export time period (which is longer than a tracking time period), the device transmits all stored telemetry metrics from the flow table to a collector.

CROSS-REFERENCE TO RELATED APPLICATION

This application is related to a co-pending U.S. Patent Application withAttorney Docket No. 000163-1018-101, entitled “SYSTEMS AND METHODS FORFLOW-BASED INBAND TELEMETRY” (filed on Jan. 9, 2020) which isincorporated by reference herein in its entirety.

BACKGROUND

The present disclosure relates to inband telemetry techniques. Inparticular, the present disclosure relates to a network device thatobtains inband telemetry data from packets and periodically reportsaggregated flow-based telemetry metrics for several smaller intervals.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1 shows an exemplary system for collecting inband telemetry data,in accordance with some embodiments of the disclosure;

FIG. 2 shows another exemplary system for collecting inband telemetrydata, in accordance with some embodiments of the disclosure;

FIG. 3 shows exemplary intervals for collecting telemetry data, inaccordance with some embodiments of the disclosure;

FIG. 4 shows an exemplary flow table, in accordance with someembodiments of the disclosure;

FIG. 5 is a flowchart of an illustrative process for collecting inbandtelemetry data, in accordance with some embodiments of the disclosure;and

FIG. 6 shows a diagram of illustrative devices for collecting inbandtelemetry data, in accordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

Network systems frequently collect telemetry data metrics (e.g.,latency, congestion, etc.) in order to monitor performance and health ofthe network. In some implementations, devices on the network may gathertelemetry metrics and report these metrics to a collector. For example,the telemetry metrics may be gathered within the control-plane of thenetwork, or by using inband telemetry (INT) techniques that allow thetelemetry data (e.g., per-hop telemetry data) to be embedded into thepackets themselves (or into duplicates of sampled packets).

To perform such telemetry functions, a network device (e.g., an INTnode) may gather network telemetry metrics by analyzing packets ithandles. The network device may, for example, identify a flow of packetsand calculate telemetry metrics based on telemetry data contained ineach packet of the flow (e.g., contained in the INT portion of theheader). The telemetry metrics may be averaged over an export timeperiod and reported to a collector at the end of such an interval. Ifthe chosen export time period is too small, however, the collectorfrequently becomes overwhelmed. If the chosen export time period is toolarge, on the other hand, the collector will not have access to granulartelemetry metrics. For example, a measurement of average latency in aflow over a large time period is not helpful when trying to pinpoint anexact time when a latency spike (e.g., a microburst) occurred.

To solve this problem, disclosed herein is a telemetry system having anetwork device that divides each export time period into several smallertime periods (tracking periods). The network device computes aggregate(e.g., averaged) telemetry metrics for each smaller tracking timeperiod, instead of computing aggregate telemetry metrics for the entireexport time period. For example, the network device may compute andlocally store a telemetry metric based on telemetry data from packets ofthe flow received during each smaller time period. At the end of eachlarger export time period, the network device forwards some or alllocally stored telemetry metrics to a collector. In this way, thecollector receives forwarded metrics at large intervals, thus preventingthe collector from becoming overwhelmed by frequent reports. Theforwarded metrics received by the collector, however, include granulartelemetry metrics for tracking time periods that are smaller than theexport time period, allowing the collector to better analyze, forexample, short-term spikes in latency.

In some embodiments disclosed herein, the network device may storecomputed flow telemetry metrics for more than one export time period.For example, the network device may locally store N computed flowtelemetry metrics (e.g., one metric for each of N tracking timeperiods), while each export time period may contain less than N trackingtime periods. In such implementations, when all stored aggregatetelemetry metrics are reported at the end of a current export timeperiod, the network device will report both: flow telemetry metricscalculated for tracking time periods that occurred during the currentexport time period; and flow telemetry metrics calculated for trackingtime periods that occurred during a preceding export time period. Thisapproach provides additional redundancy to the system, because even if areport for the previous export time period is lost, the collector willreceive some telemetry metrics for the previous export time period, whenmetrics are reported at the end of the current export time period.

FIG. 1 shows an exemplary system 100 for collecting inband telemetry(INT) data of network 140, in accordance with some embodiments of thedisclosure. System 100 may use multiple devices to collect telemetrymetrics (e.g., latency, packet path, queue length and congestion, etc.)for packets that traverse network 140 without invoking control planefunctionality. To accomplish this, system 100 may use INT aware devicesthat receive a packet and add telemetry to the packet before that packetis forwarded. INT data stored in a packet may be forwarded to collector102, which may be configured to receive telemetry data from multipledevices in network 140. In some embodiments, the collector may use thetelemetry data to analyze network performance as a whole, and to performnetwork management functions based on the analysis. For example,collector 102 may change routing policy, activate more switches,generate network performance reports, and/or perform any other networkmanagement function.

System 100 may include multiple INT-aware switches 150-166. While system100 shows switches, other packet-forwarding devices, such as hubs,routers or bridges, may also be used instead of, or in addition to,switches. INT-aware switches 150-166 may be configured to recognizepackets (e.g., packets 104, 106, 108) that include INT data (e.g., anINT header). When such a packet is received by one of switches 150-166,that switch may add telemetry data to the packet before that packet isforwarded to a next switch. For example, one of switches 150-166 may addits own address to the packet. In another example, the switch may alsoadd timestamps indicating when the packet was received by the switch andwhen it was forwarded to a next switch. One of switches 150-166 may alsoadd information regarding its queue size, and whether congestion wasexperienced when processing the received packet.

In some embodiments, one of switches 150-166 may compute one or moretelemetry metrics based on the data stored in a packet that it receives.In some embodiments, metrics are computed for every packet.Alternatively, metrics may be computed based on a certain percentage(e.g., 5%) of sampled packets. For example, switch 150 may receivepacket 104, which was previously forwarded by switches 158 and 154,where both switches 158 and 154 added INT data to packet 104. Switch 150then computes telemetry metrics based on data in the INT portion ofpacket 150. For example, switch 150 may compute latency of the last hopexperienced by that packet (e.g., hop from switch 154 to switch 150) bycomparing a timestamp indicating when packet 150 was sent by switch 154,and timestamp indicating when packet 150 was received by switch 150.Switch 150 may also compute other metrics (e.g., per hop metrics), suchas whether congestion was experienced during last hop and/or what thesize of the queue was during the hop. In some embodiments, switch 150may also compute other metrics, for example, switch 150 may determinewhat path packet 104 took (e.g., the path may include {switch 158,switch 154}).

Switch 150 may send INT data from packet 104 to collector 102. In someembodiments, switch 150 may send metrics calculated based on INT data tocollector 102. For example, such data may be sent every time a packet isprocessed. In some embodiments, collector 102 is configured to receivesuch INT data from all switches 150-166. For this reason, it's possiblecollector 102 may become overwhelmed when too much data comes in at thesame time. To overcome this problem, modified system 200 is described inFIG. 2.

FIG. 2 shows an exemplary system 200 for collecting inband telemetry(INT) data of network 204, in accordance with some embodiments of thedisclosure. In some embodiments, system 200 includes the same devices asshown in system 100. For example, network 204 may be the same as network140, device 212 may be the same as switch 150, device 210 may be thesame switch 154, device 208 may be the same as switch 158, and collector218 may be the same as collector 102. In some embodiments, each ofdevice 208, 210, and 212 may be an INT-enabled packet forwarding device(e.g., a router, a switch, or a hub).

As described above, INT-enabled devices 208-212 may examine incomingpackets to identify packets that include an INT header. FIG. 2 shows aprogress of a packet as it travels along network 204. For example, apacket 220 may enter network 204 with payload 222. In some embodiments,packet 220 may arrive from a device 202 that is not a part ofINT-enabled network 204 (e.g., packet 220 may arrive from a user deviceor from a switch with no INT capabilities.) Device 208 may then add aheader (H) to packet 220, while payload 222 remains the same.Additionally, device 208 may add INT telemetry data (M) to packet 220.In some embodiments, the telemetry data may be the same as describedabove in relation to FIG. 1 (timestamp data, congestion data, addressdata, queue-size data, etc.).

Subsequently, device 210 receives packet 224 which includes the headerand telemetry data added by device 208. Payload 226 remains the same.Upon detecting the presence of the INT header, device 210 may addadditional INT telemetry data to packet 224. Similarly, device 212receives packet 228 which includes the header and telemetry data addedby devices 208 and 210. Payload 230 remains the same. Upon detecting thepresence of the INT header, device 212 handles the packet as will bedescribed below.

As shown in FIG. 2, some or all network devices 208-212 may include aflow table for tracking network metrics that are aggregated for adetected flow. For example, flow tables may be kept by devices thatforward packets outside of INT-enabled network 204. In some embodiments,all devices of network 204 may maintain flow tables. In someembodiments, the flow table may be stored only by a last device on theedge of the INT-enabled network 204 (e.g., by device 212). In someembodiments, every INT-enabled of the INT-enabled network 204 maymaintain its own flow tables. For example, device 212 may include flowtable 214. An exemplary flow table is shown below in FIG. 3. In someembodiments, device 212 maintains a table that lists all packet flowsthat it has recently handled. For example, two packets may be determinedto belong to the same flow if they share the following data: sourceaddress, destination address, source port, destination port, andprotocol. In some embodiments, other methods to detect flow may be used(e.g., shared port and destination address only). In some embodiments,other ways to detect flows may be used (e.g., based on source addressand destination only).

For example, whenever a packet is received by device 212, the device maycheck if that packet belongs to a flow that is already present in flowtable 214. If not, device 212 may create a new entry 216 in flow table214. Device 212 then populates entry 216 with information that is basedon INT data from packet 228. For example, entry 216 may include latencydata, congestion data, path data, queue data, any other networktelemetry data or any combination of the above.

If the packet belongs to an already-existing flow entry, device 212updates the relevant flow table entry based on INT data from packet 228.For example, device 212 may store calculated aggregated statisticalvalues based on INT data from packet 228 and data from previouslyreceived packets from the same packet flow. Device 212 may calculate andstore a variety of statistical values, e.g., minimum, maximum, average,variance, jitter, standard deviation, mode, or any combination thereof.For example, for a latency metric, device 212 may calculate and store inthe flow table aggregated data for a per-hop latency of packets of theflow to which packet 228 belongs. In some embodiments, device 212calculates and stores in the one or more of: minimum latency, maximumlatency, average latency, variance of latency, jitter of latency,standard deviation of latency, mode of latency, or any combinationthereof in a flow table entry. Device 212 may then forward packet 228outside of network 204, e.g., to endpoint device 206 or to a switch thatis not INT-enabled. In some embodiments, device 212 may, in addition tostoring an aggregated metric for the flow, also store metrics derivedsolely from the last received packet of the flow.

At certain time intervals (e.g., periodically or a-periodically) device212 may forward aggregated data stored in flow table 214 to collector218. For example, device 212 may forward that data every minute or every30 seconds. In some embodiments, device 212 may forward the data ondemand. Compared to system 100, system 200 reduces the amount oftransmissions received by collector 218 from devices 208-212 becauseaggregated INT data is sent instead of data from every packet. Collector218 may then take network actions based on all received data (e.g.,generated warnings, changes in packet-forwarding policies, etc.). Insome embodiments, device 212 may in addition to the aggregated metricalso forward metrics calculated based on the last received packet of theflow.

In some embodiments, device 212 may collect flow-based aggregatedmetrics for time-tracking time periods that are smaller than a longerexport time period. For example, if an export period is 30 second-long,device 212 may collect flow-based aggregated metrics for 5-secondtracking periods or 10-second tracking periods (or periods any sizesmaller than the export period). Device 212 may use a flow table (asshown in FIG. 4) to store a plurality of aggregated metricscorresponding to respective tracking periods.

In some embodiments, device 212 may track metrics for N tracking timeperiods, where N is any integer number. For example, for each respectiveone of N tracking periods, device 212 may track one or more metricscorresponding to that period and based on data from packets of therespective tracking period. The number N may be selected manually orautomatically. Metrics for each respective time period of N trackingtime periods may be computed based on INT information from packetsreceived during that respective time period. At an end of an export timeperiod (that is larger than each tracking period) device 212 maytransmit to collector 218 all stored metrics for each of the N trackingtime periods. In some embodiments, device 212 may transmit to collector218 only some of the stored metrics for all N time periods. In oneimplementation, N tracking time periods may span more than one exportperiod. In this case, when stored metrics for all N tracking timeperiods are reported at the end of a current time period, some metricscollected during the current export time period will be reported alongwith some metrics collected during a preceding export time period.

In some embodiments, in addition to storing metrics corresponding toeach of the N tracking time periods, device 212 may also keep anaggregated metric computed for the entire export period. For example,information from stored metrics of all N tracking time periods may beaggregated together (e.g., by averaging). This export period-specificaggregate metric (e.g., latency) may be refreshed at the beginning ofeach export period. Such export period-specific aggregate metrics may bereported at the end of an export time period in addition to or insteadof stored metrics for all N tracking time periods.

FIG. 3 shows exemplary intervals 300 for collecting telemetry data, inaccordance with some embodiments of the disclosure (e.g., as describedabove in relation to FIG. 2). For example, tracking and export intervalsdepicted in FIG. 3 may be used by a device (e.g., device 212) toaggregate and report telemetry data to collector 218. However, any othertelemetry metric (e.g., queue size, congestion, path) may be tracked inthe same manner.

The device may track and store a telemetry metric (e.g., average per-hoplatency metric) aggregated from data (e.g., INT data) contained inpackets received during each tracking period 304-308. For example, thedevice may compute and store average latency for packets received duringtracking period 304, and then compute and store average latency forpackets received during tracking period 306, etc. In some embodiments,the device may store average latency for up to N tracking periods 302(e.g., for 12 tracking periods). For example, a respective averagelatency metric may be started for each respective tracking period of Ntracking periods 302. N may be any integer number. In such embodiments,when N+1^(st) telemetry metric is calculated, the device may store thenew metric but delete the oldest stored telemetry metric. In this way,no more than N telemetry metrics (each metric corresponding to one ofthe N tracking periods) are stored at any time. In some embodiments,this data may be tracked per flow and stored in a flow table as shown inFIG. 4.

At an end of each time period 312-316, the device may report to acollector (e.g., collector 218) all metrics (e.g., average latencymetrics) stored in the memory of the device. For example, at the end ofexport time period 312, the device may report to the collector metricscomputed for 6 tracking periods that occurred during export time period312. In another example, at the end of export time period 314, thedevice may report to the collector metrics computed for 12 trackingperiods that occurred during export time period 312 and export timeperiod 314. In this way, because metrics are reported that werecollected across more than one export period, a measure of redundancy isachieved. For example, if transmission that occurred at the end ofperiod 312 was lost, the transmission that occurred at end of period 314may be used to recover some of the lost data. In some embodiments, thedevice may include time stamps along with each reported telemetry metricto uniquely identify the tracking period for which the metric is beingreported.

Length of tracking period 310 (T) may be computed based on the length ofthe export period 318 (E), the maximum number of metrics that may bestored at the same time 302 (N) and based on a degree of overlap 320(M). As show, the degree of overlap (M) is equal to 2 (meaning thatredundancy data will be provided across two export time periods).However, any other integer number may be used. For example, a value ofM=1 will result in no overlap and no redundancy, while a value of M=3will result in overlap across three export time periods. In someembodiments, length of tracking period 310 (T) may be computed accordingto formula T=(M*E)/N. For example, as shown, N=12, and M=2. In thisscenario, if the length of the export time period (E) is 30 seconds, Ncan be computed to be equal to (2*30)/12=5 seconds. However, any valueof M, N, and E can be used.

In addition to or instead of reporting telemetry metrics for trackingperiods 304-308, the device can also report telemetry metrics for theentire export period. For example, at the end of export period 312,control circuitry may report the average per-hop latency of all packetsreceived during period 312.

In some embodiments, the tracking metrics are stored only for trackingperiods that are considered active (e.g., time periods during which atleast one packet was received). For example, if no packets were receivedfor tracking period 306, no telemetry metric (e.g., latency) will bestored for that tracking period, and that tracking period will not counttoward the limit of N actively stored time periods.

FIG. 4 shows an exemplary flow table 400, in accordance with someembodiments of the disclosure. For example, flow table 400 may be flowtable 214 stored in memory of an INT enabled network device (e.g.,device 216). Table 400 is shown with three packet flows (1, 2, and 3),however, any number of flows may be stored.

Each row of the flow table 400 represents a flow tracked by device 216.Column 402 may contain flow IDs of each flow. Rows 404-412 may identifyeach flow by source address, destination address, source port,destination port, and protocol. In some embodiments, the device maytreat any packet that includes the same source address, destinationaddress, source port, destination port, and protocol as belonging to thesame flow. For example, if a device were to receive a packet withmetadata indicating a source address to be 216.3.128.12, destinationaddress to be 127.0.0.1, source port to be 80, destination port to be 80and protocol to be TCP/IP, the device will determine that that packet ispart of flow “1.” If a packet is received that is not part of anexisting flow, a new flow entry may be created, e.g., as row 4.

Flow table 400 may also include aggregated INT statistics for each flowfor a certain number of tracking periods (e.g., N tracking periods). Forexample, column 414 may track average per-hop latency of the flow for afirst tracking period (e.g., period 304). Whenever the device receives apacket of flow 1 during the first tracking period, the column 414/row 1entry may be updated to include data from the newly received packet. Forexample, if flow “1” has average per-hop latency of 15 ms (over 2packets) during the first tracking period, and then the device receivesa packet belonging to flow “1” and indicating per-hop latency of 30 msduring the first tracking period, the column 414/row 1 entry may beupdated with a new average to be 20 ms (over 3 packets). In someembodiments, each cell in some or all of columns 414-418 may includemultiple values (e.g., separate values for each hop). For example, flow“1” may include 2 hops, and latency information may be stored (e.g., incolumn 414) separately for “hop 1” of flow “1” and for “hop 2” of flow“1.” In another example, flow “2” may include 3 hops, and congestioninformation may be stored (e.g., in column 418) separately for “hop 1”of flow “2” for “hop 2” of flow “2,” and “hop 3” of flow “2.”

Similarly, the average latency of packets of flow 1 may be tracked for asecond tracking period (e.g., period 306) in column 416, and for Nthtime period (e.g., period 308) in column 418. In some embodiments,columns may be recycled. For example, when the device needs to tracklatency metric for N+1^(st) tracking period, the device may overwritedata stored in column 414 to store the average latency for the N+1^(st)tracking period. In some embodiments, cells of table 400 may alsoinclude a timestamp identifying the tracking time period. Averageper-hop latency may be similarly tracked for tracking periods 1−N forflow 2 and flow 3. In addition to latency, any other latency telemetrystatistic (e.g., minimum latency, maximum latency, variance of latency,jitter of latency, standard deviation of latency, mode of latency, orany combination thereof) may also be stored in table 400 for each flowfor a plurality of tracking periods. Furthermore, other telemetrymetrics (congestion, path, queue size) may also be stored in table 400.

The content of flow table 400 may be periodically reported to thecollector (e.g., collector 218). For example, content of flow table 400may be reported at an end of each export period (e.g., periods 312-316).In some embodiments, flow table 400 is not cleared after an export andwill always retain data for N tracking periods (e.g., flow table 400 mayretain N respective metrics, each corresponding to one of N trackingperiods). In some implementations, content of flow table 400 may beaccessed on demand by command line interface call (e.g., from a userdevice). For example, when a user issues issue a command to the devicevia a command line, the device may return all or some of the data inflow table 400.

FIG. 5 is a flowchart of an illustrative process for collecting inbandtelemetry data, in accordance with some embodiments of the disclosure.For example, process 500 may be performed by control circuitry of adevice (e.g., a packet-forwarding device or a network device). Forexample, the control circuitry may be control circuitry 604 of networkdevice 602 of FIG. 6 as described below.

A process 500 for processing telemetry data begins at block 502, wherecontrol circuitry receives a packet. For example, a packet may bereceived using a network interface (e.g., interface 612). The packet maybe received from an outside device or from an INT-enabled device andinclude INT data.

At 504, the control circuitry processes data from the packet received atstep 502 (e.g., INT data) and updates metrics for the current trackingperiod (which is smaller than an export period). For example, thecontrol circuity may update latency for a flow to which the packetbelongs for the current time period (e.g., as show in table 400) basedon per-hop latency data of the packet.

At 506, control circuitry may check whether the current tracking periodended and a next one has begun. If not, the control circuitry returns to502 and continues to receive and process packets for the currenttracking period. If the current tracking period has ended, the devicestores the metric computed for the current tracking period at step 508.For example, the new metric is stored in table 400 which is stored inmemory (e.g., memory 606) of the device. Optionally, the controlcircuitry may check if more than a set number (N) of metrics has alreadybeen stored at step 510. If so, process 500 proceeds to step 512, wherethe oldest stored metric is deleted to make room for the new metric. Inthis way, no more than N metrics (e.g., one metric for each of Ntracking time periods) may be stored at any one time.

At 514, the control circuitry enters the new tracking period and setsthe new tracking period as the current tracking period. At 516, thecontrol circuitry also checks if an end of an export time period isreached. If not, the control circuitry returns to step 502 and continuesto receive and process packets to calculate a new telemetry metric for anew tracking period designated as current. If end of the export timeperiod is reached, process 500 proceeds to step 518.

At 518, the control circuitry may report all stored metrics to acollector (e.g., collector 218). In some embodiments, the controlcircuitry may also report an aggregate metric for the entire export timeperiod. In some embodiments, the stored metrics are not cleared afterthe report is transmitted. Instead, some of the old metrics (which arenot deleted) may be reported again at an end of the next export timeperiod to provide redundancy.

FIG. 6 shows a diagram of illustrative devices of a system 600 thatincludes network device 602, network devices 614-616, and collector 652.For example, device 602 may be the same as device 212, network device614-616 may be the same as devices 208-210, and collector 652 may be thesame as collector 218.

Device 602 may receive and send data via an input/output (I/O) path 610.I/O path 610 is communicatively connected to control circuitry 604,which includes processing circuitry 608 and storage (or memory) 606.Control circuitry 604 may send and receive commands, requests, and othersuitable data using I/O path 610. I/O path 610 may connect controlcircuitry 604 (and specifically processing circuitry 608) to one or morenetwork interfaces 612, which in turn connect device 602 to otherdevices on the network (e.g., network 204 or 140).

Control circuitry 604 may be based on any suitable processing circuitry,such as processing circuitry 608. As referred to herein, processingcircuitry should be understood to mean circuitry based on one or moremicroprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core,octa-core, or any suitable number of cores). In some embodiments,processing circuitry is distributed across multiple separate processorsor processing units, for example, multiple of the same type ofprocessing units (e.g., two INTEL CORE i7 processors) or multipledifferent processors (e.g., an INTEL CORE i5 processor and an INTEL COREi7 processor). In some embodiments, control circuitry 604 executesinstructions stored in memory (i.e., storage 606). For example, theinstructions may cause control circuitry 604 to perform packetforwarding and INT operations described above and below.

Memory 606 may be an electronic storage device that is part of controlcircuitry 604. As referred to herein, the phrase “electronic storagedevice” or “storage device” should be understood to mean any device forstoring electronic data, computer software, instructions, and/orfirmware, such as random-access memory, hard drives, optical drives,solid state devices, quantum storage devices, or any other suitablefixed or removable storage devices, and/or any combination of the same.Nonvolatile memory may also be used. The circuitry described herein mayexecute instructions included in software running on one or more generalpurpose or specialized processors.

Control circuitry 604 may use network interface 612 to receive andforward packets to other network devices 614-616 (which may includehardware similar to that of device 602), e.g., over any kind of a wiredor wireless network. In some embodiments, devices 602, 614, and 616 maybe INT-enabled device. For example, memory 606 may include instructionsfor handling INT packets to collect and forward telemetry data asdescribed above. In some embodiments, network device 602 may store aflow table in memory 606, where the flow table is established andupdated as described above. Control circuitry may periodically forwarddata from the flow table to collector 652.

Collector 652 may include I/O path 660, network interface 662, andcontrol circuitry 654 that includes processing circuitry 658 and storage656. These elements may function similarly to elements 604-612 asdescribed above. Collector 652 may be configured to receive and processtelemetry data from all devices 602, 614, and 616 via network interface662. In some embodiments, collector 652 may process all received INTdata and use that data to make network-wide actions and generatereports.

It will be apparent to those of ordinary skill in the art that methodsinvolved in the present invention may be embodied in a computer programproduct that includes a computer-usable and/or -readable medium. Forexample, such a computer-usable medium may consist of a read-only memorydevice, such as a CD-ROM disk or conventional ROM device, or arandom-access memory, such as a hard drive device or a computerdiskette, having a computer-readable program code stored thereon. Itshould also be understood that methods, techniques, and processesinvolved in the present disclosure may be executed using processingcircuitry.

The processes discussed above are intended to be illustrative and notlimiting. More generally, the above disclosure is meant to be exemplaryand not limiting. Only the claims that follow are meant to set bounds asto what the present invention includes. Furthermore, it should be notedthat the features and limitations described in any one embodiment may beapplied to any other embodiment herein, and flowcharts or examplesrelating to one embodiment may be combined with any other embodiment ina suitable manner, done in different orders, or done in parallel. Inaddition, the systems and methods described herein may be performed inreal time. It should also be noted, the systems and/or methods describedabove may be applied to, or used in accordance with, other systemsand/or methods.

What is claimed is:
 1. A method for network telemetry, the methodcomprising: configuring, by a network device, a flow table that stores aplurality of telemetry metrics of a packet flow, wherein each respectivetelemetry metric of the plurality of telemetry metrics: (a) correspondsto a respective tracking time period of a first plurality of trackingtime periods, and (b) is calculated based on inband telemetry (INT) dataof packets of a packet flow received during the respective tracking timeperiod; receiving, by the network device, a packet of the packet flowduring a particular time period of the first plurality of tracking timeperiods; updating the flow table based on INT data of the receivedpacket to modify a telemetry metric of the plurality of telemetrymetrics, the telemetry metric corresponding to the particular timeperiod; and at an end of an export time period, forwarding to an INTcollector the plurality of telemetry metrics of the flow table, whereinthe export time period comprises a second plurality of tracking timeperiods.
 2. The method of claim 1, wherein a given telemetry metric ofthe plurality of the telemetry metric is: per hop minimum latency, perhop maximum latency, per hop average latency, per hop variance oflatency, per hop jitter of latency, per hop standard deviation oflatency, per hop mode of latency, or path metric.
 3. The method of claim1, wherein each respective telemetry metric of the plurality oftelemetry metrics corresponds to one respective time period of Ntracking time periods, the method further comprising: receiving, by thenetwork device, a new packet of the packet flow, the new packet beingreceived during a new time period that follows the N tracking timeperiods; deleting the oldest telemetry metric of the plurality oftelemetry metrics from the flow table; calculating a new telemetrymetric based on INT data of the new packet; and storing the newtelemetry metric in the flow table.
 4. The method of claim 3, whereinthe second plurality of tracking time periods of the export time periodcomprises N tracking time periods.
 5. The method of claim 3, wherein thesecond plurality of tracking time periods of the export time periodcomprises less than N tracking time periods.
 6. The method of claim 5,wherein length of each tracking time period (T) of the first pluralityof tracking time periods is calculated according to a formula T=(M*E)/N,wherein M is a degree of overlap, and E is length of the export timeperiod.
 7. A method for network telemetry, the method comprising:receiving, by a network device, packets of a packet flow during aplurality of export time periods, wherein each export time periodcomprises a plurality of tracking time periods; calculating a respectivetelemetry metric for each of the plurality of tracking time periods ofthe plurality of export time periods, wherein each respective telemetrymetric is calculated based on telemetry data from packets of the packetflow that were received during a respective tracking time period;storing the calculated respective telemetry metrics; and forwarding, atan end of a current export time period of the plurality of export timeperiods, the stored telemetry metrics to a collector.
 8. The method ofclaim 7, wherein a given telemetry metric is per hop minimum latency,per hop maximum latency, per hop average latency, per hop variance oflatency, per hop jitter of latency, per hop standard deviation oflatency, per hop mode of latency, or path metric.
 9. The method of claim7, wherein storing the calculated telemetry metrics comprises storing Ntelemetry metrics, the method further comprising: in response tocalculating a new telemetry metric for a new tracking period: deletingoldest stored telemetry metric of an old tracking period; and storingthe new telemetry metric.
 10. The method of claim 9, wherein the Nstored telemetry metrics only comprise telemetry metrics of the currentexport time period.
 11. The method of claim 9, wherein the N storedtelemetry metrics comprise a plurality of flow telemetry metrics of thecurrent export time period and a plurality of telemetry metrics of anexport time period preceding the current export time period.
 12. Themethod of claim 9, wherein the N stored telemetry metrics comprisepacket flow telemetry metrics calculated for tracking time periodsduring which at least one packet of the packet flow was received. 13.The method of claim 9, wherein length of each tracking time period (T)is calculated according to a formula T=(M*E)/N, wherein M is a degree ofoverlap, and E is length of each export time period.
 14. The method ofclaim 7, wherein the network device is an inband telemetry (INT)-enablednetwork device.
 15. A network device comprising: memory configuring tostore a flow table, the flow table comprising a plurality of telemetrymetrics of a packet flow, wherein each respective telemetry metric ofthe plurality of telemetry metrics: (a) corresponds to a respectivetracking time period of a first plurality of tracking time periods, and(b) is calculated based on inband telemetry (INT) data of packets of apacket flow received during the respective tracking time period; acontrol circuitry configured to: receive, during a particular timeperiod of the first plurality of tracking time periods, a packet of thepacket flow; update the flow table to modify a telemetry metric of theplurality of telemetry metrics based on INT data of the received packet,the telemetry metric corresponding to the particular time period; and acommunication circuitry configured to at an end of an export timeperiod, forward to an INT collector the plurality of telemetry metricsof the flow table, wherein the export time period comprises a secondplurality of tracking time periods.
 16. The network device of claim 15,wherein a given telemetry metric of the plurality of the telemetrymetric is per hop minimum latency, per hop maximum latency, per hopaverage latency, per hop variance of latency, per hop jitter of latency,per hop standard deviation of latency, per hop mode of latency, or pathmetric.
 17. The network device of claim 15, wherein each respectivetelemetry metric of the plurality of telemetry metrics corresponds toone respective time period of N tracking time periods, and wherein thecontrol circuitry is further configured to: receive, during a new timeperiod that follows the N tracking time periods, a new packet of thepacket flow; delete the oldest telemetry metric of the plurality oftelemetry metrics from the flow table; calculate a new telemetry metricbased on INT data of the new packet; and store the new telemetry metricin the flow table.
 18. The network device of claim 17, wherein thesecond plurality of tracking time periods of the export time periodcomprises N tracking time periods.
 19. The network device of claim 17,wherein the second plurality of tracking time periods of the export timeperiod comprises less than N tracking time periods.
 20. The networkdevice of claim 19, wherein length of each tracking time period (T) ofthe first plurality of tracking time periods is calculated according toa formula T=(M*E)/N, wherein M is a degree of overlap, and E is lengthof the export time period.